Editor’s note: Dr. Dan Freedman is providing an extra bonus post today about a poor quality study that went viral late last week. Dr. Gorski’s post will publish on Monday as scheduled, just a little bit later than usual. Think of Dr. Freedman’s post as a warm-up, as Dr. Gorski plans on discussing the same study in his usual inimitable fashion. (It deserves two posts.)
There is a new preprint study of COVID-vaccine-associated myocarditis (C-VAM) by Hoeg et al. being shared on social media and several news sites that has not passed peer review. The claim from the study authors is that they replicate the Advisory Committee on Immunization Practice’s (ACIP) analysis of C-VAM event reports from the Vaccine Adverse Events Reporting System (VAERS) and found a much higher incidence than previously reported. The study falls short of this claim in many ways but perhaps most significantly, it appears the authors wandered into the VAERS quagmire with no understanding of how the system works.
Dumpster diving in VAERS has been well covered by Science Based Medicine for 13 years. This is a classic antivaccine trope where reports from VAERS are dredged up to scare people. This is despite multiple disclaimers that you have to click through to access this data, which explicitly warn against using the data as Hoeg et al. do. This is just part of the disclaimer:
VAERS accepts reports of adverse events and reactions that occur following vaccination. Healthcare providers, vaccine manufacturers, and the public can submit reports to the system. While very important in monitoring vaccine safety, VAERS reports alone cannot be used to determine if a vaccine caused or contributed to an adverse event or illness. The reports may contain information that is incomplete, inaccurate, coincidental, or unverifiable. In large part, reports to VAERS are voluntary, which means they are subject to biases. This creates specific limitations on how the data can be used scientifically. Data from VAERS reports should always be interpreted with these limitations in mind.
The ACIP has an additional advantage that these investigators lack – the raw data from these reports has been adjudicated by VAERS staff at the Centers for Disease Control (CDC). These significant limitations of the Hoeg et al analysis should be kept in mind as we review some of the individual cases. The authors claim to use the same methodology as the ACIP review but a brief review raises some suspicions. VAERS ID 1345283 describes a teen with chest pain and right axis deviation on EKG. The report states “no clear diagnosis but a suggestion that it sounded clinically like a viral pericarditis”. Right axis deviation is not one of the criteria used by the ACIP to determine cases of myocarditis or pericarditis (see Table 1). This is the problem with just plugging in search terms (“troponin”, “myocarditis”, etc) to VAERS and not thoroughly reviewing cases. As Ryan Marino said, “this is like thinking that a search for ‘gunshots’ on NextDoor is a way to track gun violence”.
The most glaring examples of cases that were not reviewed in detail by Hoeg et al are the cases with a comorbid infection. This represents a significant confounding variable which makes it impossible to discern with such limited data if the myocarditis was due to the vaccine or the intercurrent illness. VAERS ID 1334617 describes a positive SARS-CoV-2 PCR and VAERS ID 1361923 describes a rhinovirus/enterovirus positive PCR. The authors also include a report of a patient with EBV-positive PCR, serologies pending. These cases were likely excluded by ACIP due to these confounders.
There are other notable cases like VAERS ID 1382338 where the patient is described as encephalopathic to the point of needing intubation for airway protection. Is this a case of C-VAM or a viral infection causing both encephalitis and myocardial injury? VAERS ID 1386269 describes a patient with difficulty walking due to neurological weakness. No mention of any cardiac diagnosis. These cases were likely excluded by ACIP due to incomplete information.
The authors also included a 14-year-old patient who appears to have received their Pfizer vaccine before the Emergency Use Authorization (EUA) for 12-15 year olds; VAERS ID 1292713 received a 2nd dose on 4/28/21 and the EUA did not occur until 5/10/21. There is no explanation for this in the manuscript. Perhaps these dates were recorded incorrectly but without the ability to investigate this inconsistency in the data, the authors could not possibly know anything more than speculation.
There are also a number of patients with mild elevations in troponin (0.12-0.23) and mild symptoms (one patient had 1 hour of chest pain and saw their pediatrician a week later). While I lack the expertise to weigh in on what is and what is not pediatric myocarditis, there are fortunately others (twitter thread here and here) with this expertise. These pediatric cardiologists have seen patients with V-CAM and the outcomes appear to be significantly better than myocarditis or multisystem inflammatory syndrome in children (MIS-C) from the virus itself. The preprint briefly touches on this but only discusses three references for outcomes of V-CAM when there are at least four additional references (here, here, here), including the most comprehensive review to date of 63 pediatric V-CAM cases in Pediatrics, that are curiously absent from Hoeg et al.
These are just seven cases out of 257 that I and others found on brief review. It is likely that there are more, which is one of the reasons why experienced experts in the field warn that VAERS isn’t suitable for this kind of analysis. As a passive reporting system, VAERS can serve as an early signal detector that can then be evaluated in larger, active safety monitoring systems like the Vaccine Safety Database (VSD).
Despite my frustration with the authors for getting in over their heads, I am grateful that they ported the VAERS data into their own interactive database (though this should have still contained a disclaimer about the potential for misuse of this data). One of the cases (ID 1231560) was written by a mom describing the scary experience of her child being in the cardiac unit and waiting for test results. This serves as a good reminder that myocarditis is a frightening disease. We need to inform parents of this risk to the best of our abilities. However, this is not an excuse to come up with half-baked analyses including inappropriate data that will certainly be co-opted by the antivaccine movement. And when the paper fails to be published in a journal because of the above methodological flaws, the antivaccine movement will cite this as evidence of a conspiracy.
Peer review is an important academic process where many of the mistakes noted above could have been recognized and addressed. Even if the study was flawless, this type of analysis would still be problematic given the very nature of VAERS. If the authors wanted to calculate the incidence of V-CAM, they could have requested data from VSD.
The study provides a lot of important lessons on what not to do. Don’t do research in an area that is outside of your expertise without help from someone more experienced. Don’t ignore important disclaimers on the CDC’s dataset. Don’t post your methodologically-flawed study on a preprint server when you fail to find a journal. Consider that the problem is with your analysis, not the journals. Don’t dumpster dive in VAERS. Many people, including Hoeg et al., will not understand the complexity and flaws of a useful warning system like VAERS. If you do not have access to all of the data, don’t let assumptions take the place of limitations.
VAERS is like when your uncle is in the hospital but your aunt won’t let you talk directly to the healthcare professionals so that all you get is filtered, incomplete and sometimes wildly inaccurate information. VAERS is only as good as the reporter filling out the event – and frequently the data is imperfect. That’s why the data comes with a lengthy disclaimer. Please pay attention to it.